跳转至

文章背景与核心概要

在线自蒸馏（On-policy self-distillation）是训练大语言模型的一项强大技术，但它往往受限于对简单的库尔贝克-莱布勒（KL）匹配的依赖，这在长期训练中容易导致训练不稳定和性能下降。本文引入了一种名为基于偏好的自蒸馏（Preference-Based Self-Distillation, PBSD）的新型框架，它通过奖励正则化的目标函数重新构建了自蒸馏过程。

通过优化教师样本和学生样本之间的偏好差距，而不是执行直接的分布匹配，PBSD 实现了卓越的稳定性和性能。作者提供了形式化的统计分析，证明了这种方法在何时能够优于传统的外部教师蒸馏，并在数学推理和工具使用任务中展示了其有效性。

摘要

Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization

Authors: Xin Yu, Liuchen Liao, Yiwen Zhang, Yingchen Yu, Lingzhou Xue, Qinzhen Guo
arXiv: 2605.05040 [cs.LG]
Submitted: 6 May 2026 (v1), 21 Aug 2026 (v2)

Summary

On-policy self-distillation is a powerful technique for training language models, yet it is often limited by a reliance on simple Kullback-Leibler (KL) matching, which can lead to training instability and performance degradation over time. This paper introduces Preference-Based Self-Distillation (PBSD), a novel framework that reframes self-distillation through a reward-regularized objective. By optimizing preference gaps between teacher and student samples rather than performing direct distribution matching, PBSD achieves superior stability and performance. The authors provide a formal statistical analysis to justify when this approach outperforms traditional external teacher distillation, demonstrating its effectiveness in mathematical reasoning and tool-use tasks.

核心贡献

Key Contributions

  • 奖励正则化目标函数： 超越了固定教师的 KL 匹配，推导出一个解析最优解为奖励加权教师分布的目标函数，从而确保学生策略在数学上被证明优于教师。
    • 偏好学习框架： 实现了一种实用的优化策略，专注于教师样本和学生样本之间的偏好差距，同时保持了在线采样的效率。
    • 理论基础： 提供了统计学分析，确立了在线自蒸馏比向外部教师学习更有效的条件。
    • 经验性能： 在多个模型规模和基准测试中，展示了训练稳定性和推理性能的持续改进。
  • Reward-Regularized Objective: Moves beyond fixed-teacher KL matching by deriving an objective whose analytic optimum is a reward-reweighted teacher distribution, ensuring the student policy is provably superior to the teacher.
  • Preference-Learning Framework: Implements a practical optimization strategy that focuses on preference gaps between teacher and student samples while maintaining on-policy sampling efficiency.
  • Theoretical Grounding: Provides a statistical analysis establishing the conditions under which on-policy self-distillation is more effective than learning from an external teacher.
  • Empirical Performance: Demonstrates consistent improvements in training stability and reasoning performance across multiple model scales and benchmarks.

获取与资源

Access & Resources

引用

Citation

如果您使用了这项工作，请通过其 arXiv 标识符引用该论文：arXiv:2605.05040 [cs.LG]。

If you use this work, please refer to the paper via its arXiv identifier: arXiv:2605.05040 [cs.LG].